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Abstract: Despite increased development and dissemination, there has been 
very little empirical research on Open Educational Resources (OER). Teachers 
and students involved in a large-scale OER initiative at eight community 
colleges across the United States were given a detailed questionnaire aimed at 
uncovering their perceptions of the cost, outcomes, uses and perceptions of 
quality of the OER used in their courses. Teachers and students alike reported 
significant cost savings and various pedagogical and learning impacts due to the 
implementation of OER in the classroom. In addition, most students and 
teachers perceived their OER to be at least equal in quality to traditional 
textbooks they had used in the past. Implications for further research are 
discussed. 
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Introduction 

Global efforts have aimed to create open educational resources (OER) for the 
purpose of improving access to education (D'Antoni, 2009; OECD, 2007). OER 
have the potential to increase learning by increasing the opportunity for 
individuals to access and interact with educational resources (Cape Town Open 
Education Declaration, 2007; National Knowledge Commission, 2007). OER take 
on various shapes and sizes including the creation of opencourseware at MIT 
and several other universities (see http://www.ocwconsortium.org), learning 
objects and modules like those made available by Connexions (see 
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http://cnx.org), openly licensed textbooks such as those offered by CK12 and, 
formerly, Flat World Knowledge (http://ckl2.org; 

http://www.flatworldknowledge.com; Bliss, Hilton, Wiley, & Thanos, 2013; 
Hilton & Wiley, 2011;), openly available classes (Fini, 2008), and Massively 
Open Online Courses (MOOCs) (Mackness, Mak & Williams 2010; Fini, 2009). 
More recent developments include Udacity (http://udacity.com/), Coursera 
(http://coursera.org/), and EdX (http://www.edxonline.org/), which intend to 
make learning resources freely available and provide low-cost certification as 
well. 

While OER provide new opportunities for access to educational resources, many 
also see in OER an opportunity for students and school districts to save 
substantial amounts of money by eliminating the need to purchase expensive 
textbooks (Bliss, et al, 2013; Hilton & Wiley, 2011). For example, Caswell 
(2012) describes a major curriculum redesign initiative undertaken by the 
Washington State Board of Community and Technical Colleges to leverage OER 
and original, faculty-authored content. The intended result is that professors will 
be able to use OER to replace traditional textbooks, dramatically lowering the 
cost that students pay for their education. 

While much work has been done to create, disseminate, and champion OER, 
relatively little theory-based, generalizable research has been done to examine 
its use and impacts (Bateman, Lane & Moon 2012). Vital questions about many 
aspects of OER still need to be addressed, which we view through the lens of a 
"COUP framework" (Cost, Outcome, Use, Perceptions) encompassing the impacts 
on Cost, impact on Outcomes, Use patterns, and Perceptions of quality of OER. 
The four components of the COUP framework comprise the salient aspects of 
education that we consider most likely to be impacted by the use of OER. 
Specifically, OER could potentially affect educational costs because the expense 
of copying and distributing openly licensed materials (especially digital 
materials) can be greatly reduced compared to most copyrighted educational 
materials. Reduced educational costs, together with increased ability to keep 
OER up-to-date and access to materials on the first day of class, have the 
potential to impact student outcomes. Outcomes could also be affected by how 
teachers and students use OER in ways that differ from how they use traditional 
materials. Finally, student and teacher perceptions of OER have the potential to 
shed light on conflicting use patterns and outcomes. For these and other 
reasons, the use of a framework such as COUP provides a minimal basis for 
future comparative research on OER. 

Preliminary research has begun to explore the extent to which replacing 
traditional textbooks with OER can both decrease costs and improve outcomes 
in middle and high schools. Wiley et al (2012) examined the impact of using 
open textbooks based on materials produced by the CK-12 Foundation for three 
high school science courses. They found that open textbooks were over 50% 
less expensive than the traditional textbooks recommended for adoption, and 
that there was no negative impact on student learning outcomes as measured 
by state standardized tests. 

Despite promising early results in the K-12 space, no empirical work around the 
costs, outcomes, uses, and perceptions of open textbooks appears to have been 
done in community college settings. In a potential case of policy outpacing 
evidence, the US Department of Labor recently required open licensing of all 
educational materials produced under the Trade Adjustment Assistance 
Community College and Career Training (TAACCCT) Grant Program - a two 
billion dollar investment in OER. Given this new policy, it behooves us to 
understand how OER impact community college students and teachers from a 
COUP perspective. The current study contributes to the knowledgebase by 
providing additional detail about these four impacts of OER in the community 
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college setting. 

Methodology 

The study was conducted in the context of an open education initiative called 
Project Kaleidoscope (http://www.project-kaleidoscope.org/). This initiative is 
comprised of eight community colleges serving predominantly at-risk students. 
These colleges work together to create courses that replace traditional, 
expensive textbooks with OER. During the 2011-2012 academic year, Project 
Kaleidoscope (PK) impacted over 4,000 students across 80 teachers in a 
controlled pilot, with another 5,000 students using PK course designs and 
materials outside the controlled pilot. 

Over 80 community college teachers who used PK texts in their Winter 2012 
courses were asked to complete an online questionnaire about their impressions 
of these resources in terms of cost, outcomes, use and perceptions of quality 
(Annex A). Requests were made via email from the study authors containing a 
link to the questionnaire in early March, near the end of the first half of the 
semester for most courses. Some institutions had courses on the block schedule 
and instructors of these courses were sent email requests in mid-April. 

Additionally, all instructors were provided with a link to a student version of the 
questionnaire (Annex B) and asked to forward that link onto their students. 
Follow-up emails were sent to non-responding teachers and teachers with low 
numbers of responding students every 10-15 days through to the end of the 
semester. 

The questionnaires included multiple-choice, multiple-select, and constructed 
response items that asked instructors and students about each of the 
components of the COUP framework. For instance, teachers and students were 
asked to compare the cost, quality and patterns of use of their open textbook to 
what they normally experienced with traditional textbooks. In addition, teachers 
were asked about the impact of the open textbook adoption on their own 
preparation time and pedagogy and on their students' preparedness and 
engagement. 

Results 

Fifty-eight teachers from all eight PK institutions completed some items on the 
questionnaire. Respondents used PK texts to teach a variety of subjects, ranging 
from business fundamentals to introductory psychology. Nearly 60 percent of 
teacher-respondents were female, over 60 percent had taught at the college 
level for more than 6 years, and over 70 percent typically taught three or more 
courses each term. Only three teachers did not have a master's or doctoral 
degree. Sixty-seven percent of teachers indicated that they had taught the 
course in which PK materials were used at least three times, and all but six 
teachers had taught the course at least once. Finally, 42 percent of responding 
teachers taught at one of the eight PK institutions, with the remaining 58 
percent were spread fairly evenly across the remaining seven schools. This 
response distribution was fairly parallel to the actual distribution of teachers 
across the PK schools. 

In all, 490 students from all eight PK institutions completed the questionnaire. 
The disciplines within which students who responded to the questionnaire took 
courses are shown in Figure 1. Over 60 percent of student-respondents were 
female, and, on average, had completed 2.5 academic terms or semesters. 
Sixty-one percent of students in PK courses who responded to the questionnaire 
indicated having received a Pell grant or fee waiver, a proxy for low socio¬ 
economic status. The overall percentage of Pell eligible students in the surveyed 
population is not known, but it is likely more than 50 percent. Thirty-four 
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percent of students indicated having received loans to fund their education. 
Nearly 50 percent of student-respondents reported a college GPA of 3.0 or 
higher on a 4.0 scale. The majority of respondents came from three of the eight 
PK institutions (40 percent from one, 20 percent each from two others). The 
remaining 20 percent of respondents were spread across the other five PK 
schools in decreasing proportions. Response proportions were similar to actual 
enrollment in PK courses, with the most represented institution in our study 
having the highest proportion of students in the PK initiative and the least 
represented institution having the lowest proportion of students in PK courses. 

Figure 1: Distribution of student responses by discipline 
^Indicates a variety of different courses within a similar discipline (e.g. "math" 
includes developmental math, elementary developmental algebra, intermediate 
developmental algebra, and college algebra courses). 
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In the remainder of this section, we present the results of our study in terms of 
the COUP framework: cost, outcomes, use and perceptions of open digital 
textbooks in community college courses. 

Cost 

Teacher and student reports of traditional textbook costs were fairly well 
aligned. Teachers reported typically requiring students to spend at least $80 per 
course on required textbooks, while students reported spending between $60 
and $83 per course on such texts (Figures 2 and 3). 
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Figure 2: Distribution of teacher responses to the question "When you have 
taught this course in the past, how much have students generally been asked to 

spend on textbooks?" (n=49). 



Figure 3: Distribution of typical, student-reported spending on textbooks per 

term 

Columns indicate percent of students (n=488) reporting in each category (left 
vertical axis) and asterisks indicate the reported average number of courses 
taken per semester by students in each category (right vertical axis). 
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Likewise, teacher and student reports of spending on OER texts were very 
similar. Most teachers reported requiring students to spend less than $20 on 
texts in their PK courses and students reported spending between $12 and $17 
on average (Figures 4 and 5). However, a focus on the mean is misleading as 
most students reported spending no money at all on their OER texts. Indeed, 
the mode and median student reported spending on PK textbooks were both $0. 

Figure 4: Distribution of teacher responses to the question "How much have 

students been asked to spend on required textbooks in this course this 

semester?" (n = 50). 
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Figure 5: Distribution of student-reported spending on PK texts during the 2012 

Spring Term. 

Seventy-two percent (n=488) reported spending no money, and another six 
percent reported spending less than $20. 



$120 $140 
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Students who reported spending no money on textbooks in their PK courses 
were subsequently asked to select among various reasons for why they did not 
spend any money (Figure 6). Eighty-eight percent of these students selected 
"The texts were online and free." Interestingly, seven percent of students 
surveyed indicated they "couldn't afford to purchase texts." However, all but 
two of these students also selected "The texts were online and free," possibly 
indicating some confusion about what "affording a textbook" means. 
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Figure 6: Reasons students selected for why they didn't spend any money on 
texts in PK courses. Students were allowed to select more than one option 

(n = 348). 
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Outcomes 

We explored how open textbook adoption impacted students and teachers in 
several meaningful ways. Teachers were asked questions relating to three 
aspects of the impact of PK texts: teacher preparation time, pedagogical 
change, and student preparation. Finally, a few students shared thoughts about 
the impact of their OER textbook on their learning, though no specific question 
on this point was included in the questionnaire. 

Preparation time 

More than half of teachers reported spending more time preparing for their PK 
course compared to when they had taught the course in the past. Since some 
teachers participated in the actual development of the PK texts, all teachers 
were asked to report preparation time excluding time spent on text 
development (Figure 7). A chi-square test was used to determine whether there 
was a statistical difference in time spent depending on teacher participation in 

text development, but no significant difference was found [x 2 (2) = 1.64, p = 
.441], 
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Figure 7: Distributions of teacher-reported preparation time for the PK course in 
Spring Term 2012 compared to the same course in previous terms (n = 52). 
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Pedagogical change 

We also looked at how the flexibility and adaptability inherent in OER affected 
teachers' pedagogical practice. Seventy-five percent of teachers reported some 
change in instructional practice when they used the PK texts. Some reported 
changes were a direct result of the digital nature of many of the texts and other 
changes were an indirect effect resulting from the impact of PK texts on 
students. In the first case, many teachers reported increasing their effective use 
of technology in the classroom, while a handful reported technology getting in 
the way of teaching and learning. In the second case, teachers reported that 
their students were more engaged and interested in the material in the PK 
courses. These teachers, in turn, felt more latitude to give more assignments 
and assessments, as well as cover more content. 

Over 20 percent of teachers described the advantages of OER in terms of 
student access to materials at the very beginning of the course. For example, 
one teacher wrote, "Having the book immediately available online helps the 
class progress faster." And another teacher described how his students "are 
prepared from the beginning of class." A few teachers discussed other 
advantages of continuous online access to materials, including more student 
interest and engagement. One teacher described this advantage, saying, 
"Students are better prepared as they have access to the reading materials at 
little or no cost. Students are more engaged and have more interesting 
questions." Another teacher explained how better access to resources affected 
her class instruction: "I am able to refer to material knowing that all students 
will have access to the same material." 

Nearly a quarter of teachers reported no change in their teaching practice as a 
result of using the OER textbook. For example, one teacher said that her 
practice changed "remarkably little...the text provides much of the same content 
as a commercial text." Another teacher noticed a change in practice, but did not 
attribute this to the OER text: "No change due to the textbook. Lots of change 
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due to my growth as a teacher." 

Another quarter of teachers mentioned changes in time spent using technology, 
for better and worse. Several teachers mentioned technology as a barrier to 
instruction and learning: "I spend a lot more time helping students access and 
navigate the website. I refer to posts I make to the website during class on a 
frequent basis, mainly because I am not convinced they are using the links I 
give them." Even more time consuming, one teacher reported having to "modify 
several assignments in order to help students without access to [the] Internet 
or [who] have problems accessing materials through the net." On the other 
hand, some teachers felt that technology helped their students become more 
interested and engaged in the course material. One teacher described how he 
was "able to use technology more; for example, I can access the online 
materials and engage all of my students during class time" while another 
teacher "love[ed] showing the video clips during the PowerPoint presentation." 

Teachers reported a wide variety of other positive changes to their practice. 
These changes were reflected in 30 percent of responses and ranged from 
giving more assignments, to more activity in class, to creation of more 
assessments and learning tools. One teacher even took the opportunity to have 
his students write an open textbook: "I've really backed off lecturing. Because 
of the lack of high-quality OER materials for biology, my students are spending 
lots of time writing their own book." Another teacher noted how much more 
aligned her instruction was with the course materials: "My weekly posts/lecture 
notes are directly related to the OER textbook. It has helped my instruction as 
far as the students using a functional resource for their papers." The positive 
pedagogical effect of OER is reflected in the comment of one late-adopting 
teacher: "I didn't integrate until the end of the semester. Had I known how 
happy and responsive my students would be to the use [of] the OER textbook, I 
would have started much earlier in the semester." 

Student preparedness 

Sixty-three percent of teachers believed that their students were equally 
prepared in the course with PK texts compared to students in the same course 
before implementation of PK texts (Figure 8). Another 26 percent felt that their 
students were more prepared than students in the same course in the past. The 
final 11 percent perceived their students as being less prepared. 
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Figure 8: Distribution of teacher responses to the question, "How did your 
students' preparedness in the course compare to previous semesters?" (n = 54). 


70% 



Student Preparedness 


Student comments on learning 

Many students commented that they were better able to learn with an online 
book. One student wrote that he found himself "using [it] a lot and learning 
better than before." Another student noted that the OER text "helped me to 
study and learn what I needed to learn within the course." Of particular note, a 
learning-disabled student wrote, "I have a learning disability and it seemed like 
I am doing better in my grades reading the texts online than in my [printed] 
books." 

At the same time, a few students felt that the OER texts impeded their learning. 
For example, one student wrote that his textbook was "not the best quality to 
learn from." Another student gave a bit more detailed explanation: 

I think the textbook is terrible. First of all, the entire book is black and white. 
The examples are good, but not having color is making the book 10X harder to 
read and understand. They have quizzes and true/false questions at the end of 
the chapter but they don't provide answers for them....how am I suppose to 
learn?! 

More examples of both positive and negative student-reported impacts on 
learning are included in a later section on students' perceptions of quality. 

Use 

Neither teachers nor students reported meaningful increases in student use of 
PK texts compared to use of traditional texts. Students were asked to report 
how frequently they used their textbooks, both typically and in the PK course 
specifically. For typical courses, nearly half of the students reported using 
textbooks two to three times per week and another quarter reported using them 
daily (Figure 9). For PK courses, slightly less than half of the students reported 
using the OER textbooks two to three times per week, but slightly more than a 
quarter reported using them daily (Figure 10). The largest difference between 
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reported typical textbook usage and PK textbook usage was in the "Never" 
category. Only one percent of students reported never using textbooks typically, 
but 12 percent reported never using the PK texts. Nearly 75 percent of students 
who reported that they never used the OER textbook were associated with just 
5 different teachers. 

Figure 9: Student responses to a question about how often they used the 
required texts in typical courses (n=484). 
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Figure 10: Student response to a question about how often they used the 
textbook in their PK courses (n=486). 
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Perceptions of Quality 

Eleven percent of teachers and six percent of students perceived the OER 
textbooks as being of worse quality than traditional textbooks they had used in 
the past. In all cases, these users cited technology problems or general poor 
text quality as reasons for giving a low rating. 

Six teachers provided a description of what they thought made the OER texts 
worse than texts in other courses. Their responses clustered in two major 
categories: technology issues and poor quality. The two teachers who mentioned 
technology issues did not provide much detail, but it appears that student 
access to the Internet and perceived student preference for printed textbooks 
was at the heart of what made the OER texts worse for these teachers. For 
example, one of the teachers wrote, "Students have limited access-want print 
sources [because] that is what they are used to." The other teacher simply 
wrote, "That it's online." 

The remaining four teachers who thought the OER texts were worse than other 
texts focused on various aspects of quality in general. One teacher explained 
that in her text the "information is disconnected [and] the images and diagrams 
are poor," another teacher mentioned that, "the layout is hard to read," and a 
third teacher described the coverage of her textbook as "not as thorough" as 
her previous book. The fourth teacher described his perceptions of the 
limitations of OER in his discipline and a creative solution he devised to 
overcome these challenges: 

"The Online Biology book is terrible. It is too detailed, poorly written, and the 
diagrams are not nice. This is why I'm writing an ecology text with students. 
[The online text] sources are either very outdated, more advanced than the 
level I teach at, and poor quality." 

In all, 20 students provided a description of what they thought made the OER 
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texts worse than other texts they had used in the past. Their responses 
clustered in the same two major categories: technology issues and text quality. 

Student comments on technology issues centred on website functionality and 
accessibility of materials in multiple formats. For example, one student 
expressed a desire for the website to "have an app for phones if the content is 
going to go through the Internet. If the content was able to be put into an 
e-reader such as the Kindle, that would be amazing as well." Another student 
wrote about how "the left and right scroll bar made the texts a nuisance to 
read." A third complained that "the website format made the text difficult to 
view." 

Most students who thought the OER texts were worse than traditional texts 
were concerned about general text quality. These student comments related to 
the quality of the content itself, alignment of the content with course 
expectations, presentation issues (like typos and grammar), and the physical 
quality of the print-on-demand OER texts. One student noted the he "didn't like 
the format. It asks a question and then answers it in the following paragraphs. 
To me it is not clear info and I have to often re-read." Another student wrote 
about her text's alignment with the course: "The info on this site for my course 
goes beyond the scope of my class and fails to break down the basics. Usually if 
I don't understand something in lecture I can go to my textbook for 
clarification. However this site failed to break down the basics for me before 
delving too deeply into the details." 

Despite these important critical comments, the majority of teachers and 
students perceived the OER textbooks as being of at least equal quality 
compared to the traditional textbooks they had used in the past. Respondents in 
this category were not asked to explain their reasoning. 

Finally, a large proportion of teachers (34 percent) and students (39 percent) 
perceived the OER textbooks as being of better quality than traditional texts. In 
total, 17 teachers provided a description of what they thought made the OER 
texts better. Their responses clustered in three major categories: cost, 
customization, and general quality. 

More than 20 percent of the positive teacher comments reflected a feeling that 
the cost of the OER textbook was better both for the students and for 
themselves. For example, one teacher said, "The materials were free to my 
students, which reduced a barrier to their chances for academic success." 
Another teacher stated, "It's very helpful because I didn't have to spend lots of 
money on the text book." 

A full one-third of teacher comments focused on the customization aspect of the 
OER textbook. They expressed appreciation at being able to tailor the book to 
their course and to their students. One teacher stated, "I put together the 
resources, so I could write the information my students receive. Further, I can 
select the best (easiest to understand and clearest written) sources for my 
students." Another teacher said the OER were "perfectly aligned with our critical 
reading course learning objectives. Finding critical reading instructional 
materials for college-level students is very difficult, and now we have 
custom-made materials that all instructors can use." 

Forty-five percent of the positive teacher comments centred on the better 
quality of the OER textbook. These comments on quality addressed the 
simplicity, relevancy, and organization of the textbook. One teacher appreciated 
the "currency and articulation of content [and the] organization of content," 
while another teacher liked that there were "no bells and whistles. [The book] 
covers all the required materials [and] presents the material in a simple, easy to 


14 of 25 


JIME: An OER COUP: College Teacher and Student Perceptions of Op... 



understand fashion." Other teachers stated that the book was well-written and 
more interesting than books they had used in the past. For example, one 
teacher responded that the book they used "is well written and has a good 
mixture of text, questions and case studies. The business communication 
textbook I used for a previous class was very dry." 

In addition, 160 students provided a description of what they thought made the 
OER texts better. Their responses clustered in six major categories: technical 
advantages, learning aides, customization/alignment, cost, access, and general 
quality/presentation. 

Most of the comments on technical advantages of PK texts focused on using a 
built-in search functionality to look for specific text. One student explained this 
advantage, saying the PK text was better "because it gives you the option to 
look for specific chapters...and find any keyword to start studying and 
reviewing." Another student mentioned that "it was easier to browse through 
and find the information that was needed." Other technical advantages included 
being able to "zoom in the size of the print" for "reading comfort" and "scroll up 
and down to any part of the book quickly." 

Nine percent of student responses focused on learning aids. In these comments, 
students talked about the usefulness of embedded videos, quizzes, visuals, and 
other study helps. One student felt that his OER text was better than traditional 
texts because it "gives video lessons which really help if you don't understand." 
Another student agreed that "the videos are...very helpful." 

Ten percent of students recognized the advantages of the customizable nature 
of the open texts. These students wrote about how their texts were better 
aligned with content covered in lecture and how the content was "more 
up-to-date" and "relevant" than other texts they had used in the past. A few 
students even recognized the pedagogical advantages from the teachers' point 
of view, as evidenced in the following student comment: "[My text] allowed the 
instructor to teach as the department intended, instead of using an outside 
book and basing the class around the book." 

Because the OER texts were offered to PK students for free, it could be 
expected that most students would cite cost as a major reason for feeling that 
their open texts were better than traditional texts. However, only 20 percent of 
students mentioned cost in their comments. For many of the students who did 
comment, the cost of their open texts seemed critical to their education. For 
example, one student said, "I have no expendable income. Without this free 
text I would not be able to take this course." Most of the students who 
mentioned cost also gave other, non cost-related, reasons for preferring the 
open texts over other texts they had used in the past. 

Over a quarter of student comments about what made their open texts better 
than traditional texts centered on convenient access. One student wrote that 
"it's easier when [my book] is online. I don't have to carry a big book. I could 
just carry my laptop and I have all the materials I need." Another student 
explained that "since [the book] is online I can access it anywhere there is an 
Internet connection." Interestingly, many students explicitly stated that having 
their texts online was better because "you don't have to carry around a heavy 
textbook" or worry about it getting damaged. 

There was also a wide variety of student comments about the overall quality of 
the content and presentation of the OER texts. Comments grouped under this 
theme related to the text being "easy to read," "interesting," and "providing 
examples." Student comments in this theme also mentioned the organization, 
layout, navigability, simplicity, concision, and clarity of the open texts compared 
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to more traditional texts. For example, one student said the textbook is, "short 
and straight to the point with follow-up questions to better one's 
understanding." Another student described his text as "organized very well and 
easy to read." In fact, the idea of the open texts being "easy" or "easier" than 
other texts was the predominate theme throughout all responses to the 
question asking students to describe what made quality of the OER texts better 
than those in other courses. 

Near the end of the survey, students were given an item that attempted to 
assess their textbook preferences. This item asked, "How likely are you to 
register for a future course that uses a textbook like the one used in this 
course?" The majority of students reported that they would be "Very Likely" (53 
percent) or "Somewhat Likely" (32 percent) to register for a future course with 
a textbook like the one used in their PK courses. Only a few reported being 
"Somewhat Unlikely" (eight percent) or "Very Unlikely" (seven percent) to 
register for such a course. 

It is important also to consider students' exposure to digital textbooks when 
interpreting this data. In all, 383 students indicated that their PK texts were 
accessed primarily in digital format (usually online). Only 93 students indicated 
that their PK texts were not accessed primarily in digital format. A chi-square 
test of independence was used to examine whether student likelihood to 
register for future OER courses depended on digital text exposure. No significant 
result was found (). The majority of student respondents (76 percent) would be 
somewhat or very likely to enroll in a future course that uses a text similar to 
that used in their PK course, regardless of whether their PK text in the 2012 
Spring term was primarily online or not. 

Discussion 

Following the COUP framework, we next discuss the results of this study in 
terms of cost, outcomes, use, and perceptions of open digital textbooks. 

One of the primary goals of Project Kaleidoscope was to dramatically decrease 
the cost students pay for textbooks. Both teachers and students in the study 
perceived that textbook costs were reduced by nearly 80 percent. Consequently, 
the project appears to have been successful in demonstrating that open 
educational resources can be used to lower the cost of attending community 
college. Future research should focus on understanding how savings only 
dropped by 80 percent when the cost of required textbooks on class syllabi was 
$ 0 . 

On the other hand, while there was general agreement that OER reduced the 
cost of educational materials to students, many teachers reported that using 
OER required more preparation time than using traditional materials. It is 
presumed that teachers spent more time preparing to teach either because they 
were adapting and revising existing OER or they were calculating in the time 
spent developing the materials specific to this initiative. Either way, this finding 
has at least one important implication for future OER initiatives: despite the 
many clear advantages of OER, obtaining instructor buy-in could be hindered by 
the reality of increased preparation time. One approach to ameliorate this 
concern would be to get buy-in first from those willing to spend more time 
preparing and then scale to instructors less willing to spend more time by 
providing them with pre-existing material. This particular approach was 
successfully used in the Project Kaleidoscope initiative, where several faculty 
members were brought on initially to develop content and then others were 
added later as "adopters." Only some of the adopters reported increased 
preparation time, indicating that the use of OER does not necessarily lead to 
more time spent preparing to teach. 
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While the cost story is quite clear, the teaching and learning outcomes of the 
project are mixed. Even after removing time spent in the upfront development 
of the OER textbook replacements, teachers still reported spending more time 
preparing for classes using these materials. This must be interpreted as a 
negative outcome of the study. However, the majority of teachers also reported 
that students using open textbooks were better prepared for class. This should 
be interpreted as a positive outcome of the study. Future research should 
determine if faculty think this tradeoff is acceptable. Do teachers spend 
significantly more preparation time in exchange for students who are only 
minimally better prepared? Do teachers spend minimally more preparation time 
in exchange for students who are significantly more prepared? 

Student use of their open textbooks appears to be unchanged from their use of 
other textbooks. This might be interpreted as a "do no harm" or neutral finding. 
Future research should probe a variety of issues relating to how students could 
be better prepared for class if their use of textbooks was unchanged. Also, how 
is it possible that students report that they appreciate the ability to keyword 
search their books, for example, and still report that their use of the books had 
not changed? 

While some teachers and students perceived the open textbooks as being lower 
quality than the traditional books they had used before, over three times as 
many teachers and six times as many students thought the books were higher 
quality than traditional books. This should be interpreted as a positive finding of 
the study, particularly in the context of recent concerns about the potential for 
production and dissemination of low quality OER (e.g., WCET, in press). 

Overall, the results of this study on the impacts of replacing traditional 
textbooks with OER in community college settings are mixed. However, these 
mixed findings reflect a single approach to using open educational resources in 
community college classes. Different approaches will likely produce results that 
are more uniformly negative or positive. We believe that the results presented 
above should be interpreted as a first effort at using OER at scale - an effort 
that can be improved upon and strengthened. 

Limitations 

There are a number of limitations to this study, including the study's reliance on 
questionnaire data. The results are only as accurate as the perceptions of those 
being surveyed. For example, we gained data about students' and teachers' 
perceptions of costs. However, an important follow-up to this study (on cost) 
would include an inspection of syllabi from the OER and non-OER versions of 
these courses to determine actual costs of required texts in both versions. 

A second general weakness of this study is that student and teacher perceptions 
about the use and quality of OER were likely coloured by their more general 
perceptions about the use of technology in education. Results about pedagogical 
impacts of OER are encouraging for proponents of such resources, but further, 
in-depth qualitative studies of these kinds of impacts are much needed. 

While discussing limitations, we must note that response rate is not a limitation 
in our study because we were not intending to conduct a survey, in the 
statistical sense, of a particular population. Our goal was to obtain a large 
amount of qualitative data rather than a statistically representative sample of 
the entire population of PK students. Our priority, then, was to obtain data from 
each of the eight participating institutions and from as many different 
instructors and students as possible. Indeed, student and faculty responses are 
quite representative of the spread of participants in PK courses across the 
institutions. For instance, most responses came from the institution with the 
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largest number of participating instructors and students. 

Finally, while our study did not explore the issue, the impact of open textbook 
adoption on student achievement is an extremely important outcome for future 
studies. Future research should look at the differences in student success rates 
across teachers and across time. The results presented above may be 
meaningless if use of open textbooks in the Project Kaleidoscope model were to 
decrease student learning. 
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Annex A 

Kaleidoscope Survey - Faculty - Spring 2012 

The following questions pertain to the course or courses you are teaching using 
open educational resource (OER) textbooks. If you are teaching multiple courses 
using OER textbooks, please respond to these questions in terms of what would 
be typical for such a course. After you have completed this survey, please 
remember to send the student survey link to your students IN ANY course you 
teach using OER textbooks. 

Q2 What is your gender? 

• Male (1) 

• Female (2) 

Q3 How long have you been teaching at the college level? 

• Less than 3 Years (1) 

• 3 - 6 Years (2) 

• 6 - 9 Years (3) 

• 9 - 12 Years (4) 

• 12 - 15 Years (5) 

• 15 - 18 Years (6) 

• More than 18 Years (7) 

Q5 What is the highest degree you have obtained? 

• High School Diploma (1) 

• Associate's Degree (2) 

• Bachelor's Degree (3) 

• Master's Degree (4) 

• Doctoral Degree (5) 

Q6 What is your average teaching load during a regular length semester at your 
institution? 

• 1 Course (1) 

• 2 Courses (2) 

• 3 Courses (3) 

• 4 Courses (4) 

• 5 Courses (5) 

• More than 5 Courses (6) 

Q7 How much did you communicate with your students about the OER textbook 
approach used in your course this semester? 

• Never(1) 
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• Once (2) 

• 2-4 Times (3) 

• 5-7 Times (4) 

• 8-10 Times (5) 

• Every Class Meeting (6) 

Q9 Including this semester, how many times have you taught this course? 

• 1 ( 1 ) 

• 2 ( 2 ) 

• 3 (3) 

• 4 (4) 

• 5 (5) 

• 6 ( 6 ) 

• 7 (7) 

• 8 ( 8 ) 

• 9 (9) 

• 10 ( 10 ) 

• More than 10 (11) 

Answer If Including this semester, how many times have you taught t... 1 Is 
Not Selected 

Q12 When you have taught this course in the past, how much have students 
generally been asked to spend on required textbooks? 

• Less than $20 (1) 

• $21 - $40 (2) 

• $41 - $60 (3) 

• $61 - $80 (4) 

• $81 - $100 (5) 

• $101 - $120 ( 6 ) 

• $121 - $140 (7) 

• More than $140 (8) 

Answer If Including this semester, how many times have you taught t... 1 Is 
Not Selected 

Q40 How much have students been asked to spend on required textbooks in 
this course THIS SEMESTER? 

• Less than $20 (1) 

• $21 - $40 (2) 

• $41 - $60 (3) 

• $61 - $80 (4) 

• $81 - $100 (5) 

• $101 - $120 ( 6 ) 

• $121 - $140 (7) 

• More than $140 (8) 

Answer If Including this semester, how many times have you taught t... 1 Is 
Not Selected 
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Qll When you have taught this course in the past, what percent of students do 
you think purchased the required textbooks? 

• Less than 10% (1) 

• 10 - 30% (2) 

• 30 - 50% (3) 

• 50 - 70% (4) 

• 70 - 90% (5) 

• More than 90% (6) 

Q13 How much time did you spend preparing to teach this course this semester 
(not counting time spent developing the textbook) compared to previous 
semesters? 

• Much Less Time (1) 

• Somewhat Less Time (2) 

• About the Same Amount of Time (3) 

• Somewhat More Time (4) 

• Much More Time (5) 

Q15 How did your students' preparedness in the course compare to previous 
semesters? 

• Students were LESS PREPARED this Semester (1) 

• Students were EQUALLY PREPARED this Semester (2) 

• Students were MORE PREPARED this Semester (3) 

Q14 How often do you think students used the OER textbook for this course 
throughout the semester? 

• Never(1) 

• 2-3 Times a Semester (2) 

• 2-3 Times a Month (3) 

• 2-3 Times a Week (4) 

• Daily (5) 

Q36 How has your classroom instruction changed as a result of using the OER 
textbook? 

Q16 What feedback did you receive from your students about the OER textbook 
used in this course? 

Q17 In future courses, how likely are you to use OER textbooks like that used in 
this course this semester? 

• Very Unlikely (1) 

• Somewhat Unlikely (2) 

• Somewhat Likely (3) 

• Very Likely (4) 

Q18 Did you work on the development of the OER textbook for this course? 

• Yes (1) 

• No (2) 
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Q19 How would you rate the quality of the OER textbook used for this course? 

• WORSE than the quality of texts in my other courses (1) 

• About the SAME AS the quality of texts in my other courses (2) 

• BETTER than the quality of texts in my other courses (3) 

Answer If How would you rate the quality of the OER textbook used f... WORSE 
than the quality of texts in my other courses Is Selected 

Q20 Please briefly describe what made the quality of the OER textbook WORSE 
than those in other courses. 

Answer If How would you rate the quality of the OER textbook used f... BETTER 
than the quality of texts in my other courses Is Selected 

Q21 Please briefly describe what made the quality of the OER textbook BETTER 
than those in other courses. 

Annex B 

Kaleidoscope Survey - Students - Spring 2012 

Q2 What is your gender? 

• Male (1) 

• Female (2) 

Q3 Have you received any LOANS to fund your education? 

• Yes (1) 

• No (2) 

Q4 Have you received any PELL GRANTS or FEE WAIVERS to fund your 
education? 

• Yes (1) 

• No (2) 

Q5 How many terms have you completed in college? 

• Less than 1 (1) 

• 1-2 ( 2 ) 

• 3-4 (3) 

• 5-6 (4) 

. 7-8 (5) 

• 9-10 (6) 

• More than 10 (7) 

Q6 What is your cumulative college Grade Point Average (GPA) on a 4.0 scale? 

• 0.0 - 1.4 (1) 

• 1.5 - 2.0 (2) 

• 2.1 - 2.5 (3) 

• 2.6 - 3.0 (4) 

• 3.1 - 3.5 (5) 

• 3.6 - 4.0 (6) 

• This is my first term (7) 
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Q7 In general, how often do you purchase the required textbooks for the 
courses you take? 

• Never(1) 

• Rarely (2) 

• About Half the Time (3) 

• Often (4) 

• Always (5) 

Q8 How much do you typically spend on textbooks each semester? 

• Less than $100 (1) 

• $101 - $200 ( 2 ) 

• $201 - $300 (3) 

• $301 - $400 (4) 

• $401 - $500 (5) 

• More than $500 (6) 

Q58 On average, how many courses do you take each semester? 

• 1 ( 1 ) 

• 2 ( 2 ) 

• 3 (3) 

• 4 (4) 

• 5 (5) 

• 6 ( 6 ) 

• 7 (7) 

• 8 ( 8 ) 

• More than 8 (9) 

Q25 For a typical course, how often do you use the required texts? 

• Never(1) 

• 2-3 Times a Semester (2) 

• 2-3 Times a Month (3) 

• 2-3 Times a Week (4) 

• Daily (5) 

Some of the questions that follow refer to "this course." In these questions, we 
are referring to the course taught by the instructor who sent you the link to this 
survey. 

Q54 If a new printed textbook costs $100 and the used version of the printed 
textbook costs $75, what do you think is a fair price for a digital (e.g. online, 
downloadable) version of the textbook? 

Qll Did you spend any money on textbooks for this course? 

• Yes (1) 

• No (2) 

Answer If Did you spend any money on textbooks for this course? Yes Is 
Selected 
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Q12 How much did you spend on textbooks for this course? 

• Less than $20 (1) 

• $21 - $40 (2) 

• $41 - $60 (3) 

• $61 - $80 (4) 

• $81 - $100 (5) 

• $101 - $120 ( 6 ) 

• $121 - $140 (7) 

• More than $140 (8) 

Answer If Did you spend any money on textbooks for this course? No Is 
Selected 

Q13 Why did you NOT spend any money on textbooks for this course? (select all 
that apply) 

• The texts were online and free (1) 

• I borrowed someone else's texts (2) 

• I used library copies (3) 

• I couldn't afford to purchase the texts (4) 

• The texts were sold out (5) 

• I rented the texts (6) 

• Other reasons(7)_ 

Q16 How often did you use the textbook for this course during the semester? 

• Never(1) 

• 2-3 Times a Semester (2) 

• 2-3 Times a Month (3) 

• 2-3 Times a Week (4) 

• Daily (5) 

Q17 How would you rate the quality of the textbook used for this course? 

• WORSE than the quality of the texts in my other courses (1) 

• About the SAME AS the quality of the texts in my other courses (2) 

• BETTER than the quality of the texts in my other courses (3) 

Answer If How would you rate the quality of the texts used for this... WORSE 
than the quality of the texts in my other courses Is Selected 

Q18 Please briefly describe what made the quality of this course's textbook 
WORSE than those in other courses. 

Answer If How would you rate the quality of the texts used for this... BETTER 
than the quality of the texts in my other courses Is Selected 

Q19 Please briefly describe what made the quality of this course's textbook 
BETTER than those in other courses. 
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Q20 Was the textbook used in this course available to you primarily in digital 
format? 

• Yes (1) 

• No (2) 

Answer If Was the textbook used in this course available to you pri... Yes Is 
Selected 

Q21 What do you think of the DIGITAL FORMAT of the textbook used for this 
course? 

• I like the digital format MORE than traditional printed texts (10) 

• I like the digital format LESS than traditional printed texts (11) 

• I have no preference (12) 

Q22 Overall, what do you think of the textbook used in this course? 

Q23 How likely are you to register for a future course that uses an online 
textbook like the one used in this course? 

• Very Unlikely (1) 

• Somewhat Unlikely (2) 

• Somewhat Likely (3) 

• Very Likely (4) 

Q24 Imagine a future course you are required to take. If two different sections 
of this course are offered by the same instructor during equally desirable time 
slots, but one section uses digital textbooks and the other uses traditional 
published textbooks, which section would you prefer to enroll in? 

• I would enroll in the section that uses a TRADITIONAL PUBLISHED 
TEXTBOOK (1) 

• I would enroll in the section that uses a DIGITAL TEXTBOOK (2) 

• I would have no preference (3) 
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